The present invention relates to techniques by which a user can control the retrieval of visual information. More specifically, the invention relates to a user interface that permits a user to select recorded visual information, such as video stills or sequences, for playback.
Tsunekawa et al., EP-A No. 237,324, describes a computer-controlled document data filing system, shown and described in relation to FIG 1. Data retrieval is performed on previously compressed images in response to keyboard signals from an operator, as shown and described in FIGS. 2A and 2B. A controlled displays questions asking for retrieval data specifying requested document data; the retrieval data may be code data such as a keyword for specifying a figure content of a desired document. A data retrieval unit then searches for document data satisfying the condition of the input retrieval data; if the requested document data is detected, the controller reads out the compression image data and reproduces an image of the retrieved document data for the operator.
Makkuni, R., "Representing the Process of Composing Chinese Temples," Design Computing, Vol. 1, No. 3, 1986, pp. 216-235 and Makkuni, R., "A Gestural Representation of the Process of Composing Chinese Temples," IEEE Computer Graphics & Applications, Vol. 7, No. 12, Dec. 1987, pp. 45-61, (hereinafter "the Chinese Temple articles") describe a computing based desing environment that employs a gestural language in its user interface. As shown and described in relation to FIGS. 4-8 of the IEEE article, the user provides gestures by making strokes with an input device such as a mouse. Each stroke is recorded as a series of points, sampled against direction. A compression algorithm receives the stroke as a path of points, producing a small collection of knot-points that, along with a spline function, can approximate the original stroke. This collection of knot-points is then decomposed into its parts by the directionality of movement in the creation of the stroke, inferring direction from the relation of each knot point to the previous and following knot points. The set of direction-parts is compared with predefined gesture types, and if a matching gesture type is found, its corresponding desing properties are applied to an image of a design. The application of the design properties may vary with a parameter of the gesture, such as its height.
Williams, M. D., "What Makes RABBIT Run?" International Journal of Man-Machine Studies, Vol. 21 (1984), pp. 333-352, describes RABBIT, a system that is based on retrieval by reformulation and that provides a user interface to aid users in formulating a query. Page 333 describes how the user interactively refines partial descriptions of a target item by criticizing successive example instances that satisfy the current partial description. Page 335 describes how a user initiates a query, in response to which RABBIT presents an instance that satisfies the description. The user can then refine the query using a pointing device such as a mouse to point at various properties of the instance presented and critiquing them. Page 336 states that RABBIT limits the critique functions presented in a pop-up menu to those acceptable to an attribute the user has indicated. Page 342 describes a map analysis function that generates a high resolution map of the piece of the world that contains the matching instances, displaying the coastline of this piece of the world along with the matching instances, and page 343 describes an extension in which a user could specify a region from an analysis map to reformulate the query. Pages 345-347 describe retrieval by reformulation in more detail.
Various graphical user interfaces for workstations provide a series of interactive images, with a selection by a user in one image leading to the next image. Examples of this include ViewPoint, available from Xerox Corporation, and the Macintosh user interface, available from Apple Computer Inc. Each of these user interfaces also enables the user to edit a graphical image, and the resulting edited image can, for example, be printed.
Systems have also been proposed that include an input device on which a user can draw a character that is then recognized. The Nestor Writer product from Nestor Incorporated is an example of software that recognizes a handwritten character and displays an image of the character as recognized.
Wilson, K.S., Palenque: An Interactive Multimedia Optical Disc Prototype for Children, Bank Street College of Education Center for Children and Technology, Working Paper No. 2, Mar. 1987, describes Palenque, a system that provides a user interface through which a user can control the presentation of views of an archeological site. As described at pages 4-5, the user manipulates a joy stick to obtain the views; a site map is also presented showing the user's current location, and the user can select a location of interest on the map and then jump directly to that location. Page 5 also describes a museum database that includes "rooms" that can be entered by making a selection from a graphic of a museum facade; within a room a user can make menu and icon selections to access information, which may be stored as video still frames, motion video, audio descriptions and music, sound effects, computer sounds, text, or computer graphics. Pages 7-8 discuss interface issues, including the use of the joystick, with the joystick button used to activate the cursor and deactivate virtual travel, or vice versa, and to make selections with the cursor.
Fisher, S. S., McGreevy, M., Humphries, J., and Robinett, W., "Virtual Environment Display System," presented at ACM Workshop on Interactive 3D Graphics, Chapel Hill, North Carolina, Oct. 1986, describes a head-mounted display system controlled by operator position, voice, and gesture. Pages 4-6 describe gesture input techniques, including gloves that transmit data about arm, hand, and finger shape and position to a computer, applications of which are to provide a three-dimensional cursor in the displayed environment and to effect indicated gestures in the synthesized or remote environment. The operator can pick-up and manipulate virtual objects in the virtual environment. Multiple windows of information and simulated control panels are positioned, sized, and activated by manipulating the virtual objects in 3-space, as shown in FIG. 12. Page 9 states that the system has the capability to display reconfigurable, virtual control panels that respond to glove-like tactile input devices worn by the operator. Sutherland, I. E., "Sketchpad, A Man-Machine Graphical Communication System," AFIPS Spring Joint Computer Conference, Vol. 23, 1963, pp. 329-346, describes a system that makes it possible for a man and a computer to converse through line drawings. Pages 329-331 describe an example of how inputs from a light pen can obtain a desired drawing by changing a displayed drawing. Pages 333-334 describe the use of the light pen.
Babbel et al., U.S. Pat. No. 4,521,870, describe an audio/video system with touch responsive function display screen. As shown and described in relation to FIGS. 1, 2, and 10, a finger's position is used to find a closest choice and performs the corresponding editing action.
Fabris et al., U.S. Pat. No. 4,516,156, describe a teleconferencing technique in which the control device includes a display with a touch sensitive screen for controlling the video sources and video matrix switch in response to touches. Menu movement is shown and described in relation to FIG. 8. Various menus are shown and described in relation to FIGS. 9A-9I.
Brand, S., The Media Lab, Viking, New York, 1987, pp. 13-14, 35-50, 137-141, and plates 5, 6, 9, 10 and 13, describes various techniques developed at MIT's Media Lab. Pages 13-14 and 141-142 and plate 5 describe Movie Manual, and electronic book of transmission repair that is accessed by moving a finger over a screen. Pages 36-37 and plate 6 describe NewsPeek, a selective electronic newspaper in which a user can change the image displayed by moving a finger across the screen. Pages 40-41 describe interactive television. Pages 49, 138, and 141 and plate 5 describe Aspen Movie Map, in which the user can interactively obtain views of a town from a videodisk. Page 138-139 and plates 9 and 10 describe Spatial Data Management System (SDMS), in which the user could browse neighborhoods of data, and Put That There, in which the user pointed and issued voice commands to communicate with a computer. Plate 13 describes a glove and a suit to make a computer sensitive to gesture.
Bolt, R. A., The Human Interface, Lifetime Learning Publications, Belmont, Calif. 1984, describes Dataland, in which pictorial and textual data are arranged in three screens, one for an overview of the whole space, one for a detailed view of some potion of the space, and one touch-screen for control. Page 12 describes how the user navigates about Dataland using a joystick. Page 13 describes touch travel by touching a desired spot on the touch-screen. Pages 19-20 describe random access to video images through the touch-screen. Pages 21-25 describe the CCA system in which a user can also zoom through ports between information spaces.
Herot, C. F., "Spatial Management of Data," ACM Transactions on Database Systems, Vol. 5, No. 4, Dec. 1980, pp. 493-514, describes the CCA system, also called SDMS, in greater detail. Pages 502-503 describe motion control with a joystick. Page 506 describes the use of videodisk data, with an icon being associated with a sequence of videodisk images, so that the user can zoom in on a particular icon to see the associated frame or sequence.
Spence, R., and Aperley, M., "Data Base Navigation: An Office Environment for the Professional," Behavior and Information Technology, Vol. 1, No. 1, 1982, pp. 43-54, describe the use of pointing, gesturing, and touching in handling information. Page 46 describes the use of pointing to select an artifact.